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Accountability  Integrity  Reliability 

Highlights 


Why  GAO  Did  This  Study 

The  past  two  years  have  seen  the 
Congress  and  DOD  take  meaningful 
steps  towards  addressing  long¬ 
standing  weapon  acquisition 
issues — an  area  that  has  been  on 
GAO’s  high  risk  list  since  1990.  This 
testimony  focuses  on  the  progress 
DOD  has  made  in  improving  the 
planning  and  execution  of  its 
weapon  acquisition  programs  and 
the  potential  for  recent  acquisition 
reforms  to  improve  program 
outcomes. 

The  testimony  includes 
observations  about  (1)  DOD’s 
efforts  to  manage  its  portfolio  of 
major  defense  acquisition 
programs,  (2)  the  knowledge 
attained  at  key  junctures  of  a 
subset  of  42  weapon  programs 
from  the  2009  portfolio,  (3)  other 
factors  that  can  affect  program 
execution,  and  (4)  DOD’s 
implementation  of  recent 
acquisition  reforms.  The  testimony 
is  based  on  the  results  of  our 
annual  assessment  of  weapon 
programs.  To  conduct  the 
assessment,  GAO  analyzed  data  on 
the  composition  of  DOD’s  portfolio 
of  major  defense  acquisition 
programs.  GAO  also  collected  data 
from  program  offices  on 
technology,  design,  and 
manufacturing  knowledge,  as  well 
as  on  other  factors  that  can  affect 
program  execution. 

GAO  has  made  numerous 
recommendations  on  weapon 
system  acquisition  in  prior  work 
but  is  not  making  any  new 
recommendations  in  this 
testimony. 


View  GAO-1 0-706T  or  key  components. 
For  more  information,  contact  Michael  J. 
Sullivan  at  (202)  512-4841  or 
suliivanm@gao.gov. 


What  GAO  Found 

While  DOD  still  faces  significant  challenges  in  managing  its  weapon  system 
programs,  the  current  acquisition  reform  environment  provides  an 
opportunity  to  leverage  the  lessons  of  the  past  and  manage  risks  differently. 
This  environment  is  shaped  by  significant  acquisition  reform  legislation, 
constructive  changes  in  DOD’s  acquisition  policy,  and  initiatives  by  the 
administration,  including  making  difficult  decisions  to  terminate  or  trim 
numerous  weapon  systems.  To  sustain  momentum  and  make  the  most  of  this 
opportunity,  it  will  be  essential  that  decisions  to  approve  and  fund 
acquisitions  be  consistent  with  the  reforms  and  policies  aimed  at  getting 
better  outcomes. 

DOD  has  started  to  reprioritize  and  rebalance  its  weapon  system  investments. 
In  2009,  the  Secretary  of  Defense  proposed  canceling  or  significantly 
curtailing  weapon  programs  with  a  projected  cost  of  at  least  $126  billion  that 
he  characterized  as  too  costly  or  no  longer  relevant  for  current  operations, 
while  increasing  funding  for  others  that  he  assessed  as  higher  priorities. 
Congress  supported  several  of  the  recommended  terminations.  DOD  plans  to 
replace  several  of  the  canceled  programs  in  fiscal  years  2010  and  2011, 
hopefully  with  new,  knowledge-based  acquisition  strategies,  because  the 
warfighter  need  remains.  The  most  significant  of  these  will  be  the  effort  to 
restructure  the  Army’s  terminated  Future  Combat  System  program.  At  the 
same  time,  however,  DOD’s  portfolio  of  major  defense  acquisition  programs 
continues  to  grow.  Between  December  2007  and  July  2009,  the  number  of 
major  defense  acquisition  programs  grew  from  96  to  102  programs.  GAO  has 
previously  reported  that  DOD  should  continue  to  work  to  balance  its  weapon 
system  portfolio  with  available  funding,  which  includes  reducing  the  number 
or  size  of  weapon  system  programs,  or  both,  and  assessing  the  affordability  of 
new  programs  and  capabilities  in  the  context  of  overall  defense  spending. 

At  the  program  level,  our  recent  observations  present  a  mixed  picture  of 
DOD’s  adherence  to  a  knowledge-based  acquisition  approach,  which  is  a  key 
for  improving  acquisition  outcomes.  For  42  programs  GAO  assessed  in  depth 
in  2010,  there  has  been  continued  improvement  in  the  technology,  design,  and 
manufacturing  knowledge  programs  had  at  key  points  in  the  acquisition 
process.  However,  most  programs  are  still  proceeding  with  less  knowledge 
than  best  practices  suggest,  putting  them  at  higher  risk  for  cost  growth  and 
schedule  delays.  A  majority  of  programs  have  also  experienced  requirements 
changes,  software  development  challenges,  or  workforce  issues,  or  a 
combination,  which  can  affect  program  stability  and  execution.  DOD  has 
begun  to  implement  a  revised  acquisition  policy  and  congressional  reforms 
that  address  many  of  these  areas.  For  example,  eight  programs  we  examined 
in  the  technology  development  phase  plan  to  test  competitive  prototypes 
before  starting  system  development  and  seven  programs  plan  to  hold  early 
systems  engineering  reviews.  If  DOD  consistently  applies  this  policy,  the 
number  of  programs  adhering  to  a  knowledge-based  acquisition  should 
increase  and  the  outcomes  for  DOD  programs  should  improve. 

_ United  States  Government  Accountability  Office 


Mr.  Chairman  and  Members  of  the  Subcommittee: 


I  am  pleased  to  be  here  today  to  discuss  the  Department  of  Defense’s 
(DOD)  management  of  its  acquisition  of  major  weapons  systems — an  area 
that  has  been  a  part  of  GAO’s  high-risk  list  since  1990 — and  the  potential 
for  recent  acquisition  reforms  to  improve  program  outcomes.  While  DOD 
still  faces  significant  challenges  in  managing  its  weapon  system  programs, 
the  past  two  years  have  seen  DOD  and  the  Congress  take  meaningful  steps 
towards  addressing  long-standing  weapon  acquisition  issues.  DOD  made 
major  revisions  to  its  acquisition  policies  to  place  more  emphasis  on 
acquiring  knowledge  about  requirements,  technology,  and  design  before 
programs  start — thus  putting  them  in  a  better  position  to  field  capabilities 
on-time  and  at  the  estimated  cost.  Congress  strengthened  DOD’s 
acquisition  policies  and  processes  by  passing  the  Weapon  Systems 
Acquisition  Reform  Act  of  2009, 1  which  includes  provisions  to  ensure 
programs  are  based  on  realistic  cost  estimates  and  to  terminate  programs 
that  experience  high  levels  of  cost  growth.  The  House  Armed  Services 
Committee  Panel  on  Defense  Acquisition  Reform  issued  its  final  report 
and  made  recommendations  on  areas,  such  as  assessing  the  performance 
of  the  defense  acquisition  system,  that  were  incorporated  into  the 
proposed  Implementing  Management  for  Performance  and  Related 
Reforms  to  Obtain  Value  in  Every  Acquisition  (IMPROVE)  Act  of  2010.2  In 
addition,  DOD  has  started  to  reprioritize  and  rebalance  its  weapon  system 
investments.  In  DOD’s  fiscal  year  2010  and  2011  budget  requests,  the 
Secretary  of  Defense  proposed  ending  all  or  part  of  at  least  a  half  dozen 
major  defense  acquisition  programs  that  were  over  cost,  behind  schedule, 
or  no  longer  suited  to  meet  the  warfighters’  current  needs.  Congress’s 
support  for  several  of  the  recommended  terminations  signaled  a 
willingness  to  make  difficult  choices  on  individual  weapon  systems  and 
DOD’s  weapon  system  investments  as  a  whole. 

While  DOD’s  acquisition  policies  and  process  may  be  headed  in  the  right 
direction,  fiscal  pressures  continue  to  build.  Notwithstanding  the  federal 
government’s  long-term  fiscal  challenges,  the  Pentagon  faces  its  own  near- 
term  and  long-term  fiscal  pressures  as  it  attempts  to  balance  competing 
demands,  including  ongoing  operations  in  Afghanistan  and  Iraq,  initiatives 


'Pub.  L.  No.  111-23. 

2H.R.  5013,  111th  Cong,  (as  received  from  the  House  and  referred  to  the  S.  Comm,  on 
Armed  Serv.,  Apr.  29,  2010). 
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to  grow  and  modernize  the  force,  and  increasing  personnel  and  health  care 
costs.  While  DOD’s  fiscal  year  2010  budget  request  started  the  process  of 
reprioritizing  acquisition  dollars  to  meet  warfighters’  most  pressing  needs, 
the  department  must  still  address  the  overall  affordability  of  its  weapon 
system  investments.  DOD  should  continue  to  work  to  balance  its  weapon 
system  portfolio  with  available  funding,  which  includes  reducing  the 
number  or  size  of  weapon  system  programs,  or  both,  and  assessing  the 
affordability  of  new  programs  and  capabilities  in  the  context  of  overall 
defense  spending. 

My  statement  focuses  on  the  progress  DOD  has  made  in  improving  the 
planning  and  execution  of  its  weapon  acquisition  programs  and  the 
potential  for  recent  acquisition  reforms  to  improve  program  outcomes.  It 
includes  observations  about  (1)  DOD’s  efforts  to  manage  its  portfolio  of 
major  defense  acquisition  programs,3  (2)  the  knowledge  attained  at  key 
junctures  of  a  subset  of  42  weapon  programs  from  the  2009  portfolio, 

(3)  other  factors  that  can  affect  program  execution,  and  (4)  DOD’s 
implementation  of  recent  acquisition  reforms.  The  testimony  is  based  on 
the  results  of  our  recently  issued  annual  assessment  of  weapon  programs.4 
To  conduct  the  assessment,  GAO  analyzed  data  on  the  composition  of 
DOD’s  portfolio  of  major  defense  acquisition  programs.  GAO  also 
collected  data  from  program  offices  on  technology,  design,  and 
manufacturing  knowledge,  as  well  as  on  other  factors  that  can  affect 
program  execution.  That  work  was  conducted  in  accordance  with 
generally  accepted  government  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate 
evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our 
audit  objectives. 


3Major  defense  acquisition  programs  (MDAP)  are  those  identified  by  DOD  that  require 
eventual  total  research,  development,  test  and  evaluation  (RDT&E),  including  all  planned 
increments,  expenditures  of  more  than  $365  million  or  procurement  expenditures, 
including  all  planned  increments  ,  of  more  than  $2.19  billion  in  fiscal  year  2000  constant 
dollars. 

4GAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  GAO-10-388SP, 
Washington,  D.C.:  March  30,  2010. 
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Observations  on 
DOD’s  2009  Major 
Defense  Acquisition 
Program  Portfolio 


In  our  2010  assessment  of  weapon  programs,  we  made  several 
observations  concerning  DOD’s  management  of  its  major  defense 
acquisition  portfolio.  First,  in  DOD’s  fiscal  year  2010  budget,  the  Secretary 
of  Defense  proposed  canceling  or  significantly  curtailing  programs  with 
projected  total  costs  of  at  least  $126  billion  that  he  characterized  as  too 
costly  or  no  longer  relevant  for  current  operations,  while  increasing 
funding  for  others  that  he  assessed  as  higher  priorities.  Congress 
supported  several  of  the  recommended  terminations  (see  table  1). 
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Table  1 :  Secretary  of  Defense’s  Fiscal  Year  2010  Budget  Recommendations 

System 

Total  estimated  cost 
(dollars  in  billions) 

Secretary’s  comments 

Congressional  action 

Recommended 

termination 

VH-71  Presidential 
Helicopter 

$13 

Plan  to  develop  options  for 
a  new  program 

Conferees  recommended  $100 
million  for  technology  capture 
that  DOD  has  budgeted  for  the 
VH-71  program. 

Combat  Search  and 
Rescue  Helicopter 

Unspecified 

Plan  to  reexamine 
requirements 

Did  not  authorize 
appropriations  for  the  program. 

Next-Generation  Bomber 

Unspecified 

Will  not  initiate  new 
development  program 
without  better 
understanding  of  the 
requirement  and 
technology 

Supported  development  of  a 

Next-  Generation  Bomber 
Aircraft,  but  did  not  authorize 
appropriations. 

Future  Combat  System- 
Manned  Ground  Vehicles 

87 

Plan  to  reevaluate 
requirements,  technology, 
and  approach  before 
relaunching  and 
recompeting  program 

Directed  Army  to  develop,  test, 
and  field  an  operationally 
effective  and  affordable  next 
generation  ground  combat 
vehicle.  Conferees 
recommended  rescission  of 
$26  million  in  existing  funding. 

Transformational  Satellite 

26 

Plan  to  buy  two  more 

AEHF  satellites  as 
alternative 

Did  not  authorize 
appropriations  for  the  program. 

Ballistic  Missile  Defense- 
Multiple  Kill  Vehicle 

Unspecified 

Plan  to  reexamine 
requirements;  no  mention 
of  new  program 

Did  not  authorize 
appropriations  for  the  program. 

Recommended  end 
of  production 

C-17 

Unspecified 

Recommended  ending 
production  at  205  aircraft 

Conferees  recommended  $2.5 
billion  for  the  procurement  of  10 
C-17  aircraft,  associated 
spares,  support  equipment,  and 
training  equipment. 

DDG-1000 

Unspecified 

Recommended  ending 
production  at  3  ships 

Did  notfund  additional  ships. 
Appropriated  $1.4  billion  for 
completion  of  third  DDG-1000. 

F-22 

Unspecified 

Recommended  ending 
production  at  187  aircraft 

Did  notfund  additional  aircraft. 
Conferees  recommended 
rescission  of  $383  million  in 
existing  funding. 

Total 

$126  billion 

Source:  GAO  analysis  of  Secretary's  April  2009  statement  on  fiscal  year  2010  budget  and  fiscal  year  2010  DOD  authorization  and 
appropriations  acts. 


Second,  DOD  plans  to  replace  several  of  the  canceled  programs  in  fiscal 
years  2010  and  2011,  hopefully  with  new,  knowledge-based  acquisition 
strategies,  because  the  warfighter  need  remains.  The  most  significant  of 
these  new  programs  will  be  the  effort  to  restructure  the  Army’s  Future 
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Combat  System  program  into  several  smaller,  integrated  programs.  Third, 
DOD’s  portfolio  of  major  defense  acquisition  programs  grew  to  102 
programs  in  2009 — a  net  increase  of  6  since  December  2007.  Eighteen 
programs  with  an  estimated  cost  of  over  $72  billion  entered  the  portfolio.6 
Not  all  of  these  programs  entering  the  portfolio  are  new  starts.  For 
instance,  the  Airborne  Signals  Intelligence  Payload,  and  the  Reaper 
Unmanned  Aircraft  System  are  two  programs  that  began  as  acquisition 
category  II  programs,6  but  their  total  research  and  development  or 
procurement  costs  now  exceed  the  threshold  for  major  defense 
acquisition  programs.  Twelve  programs  with  an  estimated  cost  of  $48 
billion,  including  over  $7  billion  in  cost  growth,  left  the  portfolio.7  These 
programs  left  the  portfolio  for  a  variety  of  reasons,  including  program 
restructure,  termination,  or  completion.  When  the  Future  Combat  System 
is  added  to  the  programs  leaving  the  portfolio,  the  total  cost  of  these 
programs  increases  to  $179  billion,  including  over  $47  billion  in  cost 
growth. 

Our  2010  assessment  did  not  include  an  analysis  of  the  cost  and  schedule 
performance  of  DOD’s  major  defense  acquisition  program  portfolio  as  a 
whole.  In  recent  years,  this  analysis  showed  that  the  cumulative  cost 
growth  on  DOD  programs  had  reached  $300  billion  (in  fiscal  year  2010 
dollars)  and  the  average  delay  in  delivering  initial  capabilities  was  22 
months.  DOD  did  not  issue  timely  or  complete  Selected  Acquisition 
Reports  for  its  major  defense  acquisition  programs  in  fiscal  year  2009  for 
the  second  consecutive  presidential  transition,  which  precluded  an 
analysis  of  the  performance  of  DOD’s  portfolio.  We  will  resume  our 
portfolio  analysis  in  next  year’s  assessment. 


6Cost  data  was  only  available  for  13  of  the  18  newly  designated  major  defense  acquisition 
programs. 

6An  acquisition  category  II  program  is  defined  as  a  program  that  does  not  meet  the  criteria 
for  an  acquisition  category  I  program  and  is  estimated  to  require  eventual  total  RDT&E 
expenditures  of  more  than  $140  million  or  procurement  expenditures  of  more  than  $660 
million  in  fiscal  year  2000  constant  dollars. 

The  estimated  cost  for  these  12  programs  is  based  on  DOD’s  December  2007  Selected 
Acquisition  Reports.  Cost  growth  was  calculated  from  the  programs’  first  cost  estimate. 
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Observations  from 
Our  Assessment  of 
Knowledge  Attained 
by  Key  Junctures  in 
the  Acquisition 
Process 


At  the  program  level,  our  recent  observations  present  a  mixed  picture  of 
DOD’s  adherence  to  a  knowledge-based  acquisition  approach,  which  is 
key  for  improving  acquisition  outcomes.  In  our  2010  assessment  of 
weapon  programs,  we  assessed  the  knowledge  attained  by  key  junctures 
in  the  acquisition  process  for  42  individual  weapon  programs  in  DOD’s 
2009  portfolio.  While  program  knowledge  is  increasing,  as  in  the  past, 
none  of  the  42  programs  we  assessed  have  attained  or  are  on  track  to 
attain  all  of  the  requisite  amounts  of  technology,  design,  and  production 
knowledge  by  each  of  the  key  junctures  in  the  acquisition  process.8 
However,  if  DOD  consistently  implements  its  December  2008  policy 
revisions  on  new  and  ongoing  programs,  then  DOD’s  performance  in  these 
areas,  as  well  as  its  cost  and  schedule  outcomes,  should  improve.  Our 
analysis  allows  us  to  make  five  observations  about  DOD’s  management  of 
technology,  design,  and  manufacturing  risks  and  its  use  of  testing  and 
early  systems  engineering  to  reduce  these  risks. 


•  Newer  programs  are  beginning  with  higher  levels  of  technology 
maturity,  but  they  are  not  taking  other  steps,  such  as  holding 
early  systems  engineering  reviews,  to  ensure  there  is  a  match 
between  requirements  and  resources.  Achieving  a  high  level  of 
technology  maturity  by  the  start  of  system  development  is  an  important 
indicator  of  whether  a  match  between  the  warfighter’s  requirements 
and  the  available  resources — knowledge,  time,  and  money — has  been 
made.9  Since  2006,  there  has  been  a  significant  increase  in  the 
percentage  of  technologies  demonstrated  in  a  relevant  or  realistic 
environment  by  the  start  of  system  development.  This  increase 
coincided  with  a  change  in  statute.  In  2006,  the  National  Defense 
Authorization  Act  included  a  provision  requiring  all  major  defense 
acquisition  programs  seeking  milestone  B  approval — entry  into  system 
development — to  get  a  certification  stating  the  program’s  technologies 


sNot  all  programs  provided  information  for  every  knowledge  point  or  had  reached  all  of  the 
knowledge  points — development  start,  design  review,  and  production  start. 

9The  start  of  system  development,  as  used  here,  indicates  the  point  at  which  significant 
financial  commitment  is  made  to  design,  integrate,  and  demonstrate  that  the  product  will 
meet  the  user’s  requirements  and  can  be  manufactured  on  time,  with  high  quality,  and  at  a 
cost  that  provides  an  acceptable  return  on  investment.  Under  the  revised  Department  of 
Defense  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System  (Dec.  8,  2008), 
system  development  is  now  called  engineering  and  manufacturing  development. 
Engineering  and  manufacturing  development  follows  materiel  solution  analysis  and 
technology  development.  For  shipbuilding  programs,  this  point  occurs  when  a  program 
awards  a  detailed  design  and  construction  contract. 
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have  been  demonstrated  in  a  relevant  environment.10  While  only  one  of 
the  six  programs  that  entered  system  development  since  2006  and 
provided  data  had  fully  mature  critical  technologies — that  is, 
demonstrated  in  a  realistic  environment,  according  to  our  criteria — all 
the  programs  had  critical  technologies  that  had  been  at  least 
demonstrated  in  a  relevant  environment.  Overall,  only  4  of  the  29 
programs  in  our  assessment  that  provided  data  on  technical  maturity  at 
development  start  did  so  with  fully  mature  critical  technologies. 

While  the  technology  levels  of  DOD  programs  entering  system 
development  have  increased,  these  programs  are  still  not  regularly 
conducting  early  systems  engineering  reviews,  which  help  ensure  there 
is  a  match  between  requirements  and  resources.  We  have  previously 
reported  that  before  starting  development,  programs  should  hold 
systems  engineering  events,  such  as  the  preliminary  design  review,  to 
ensure  that  requirements  are  defined  and  feasible  and  that  the 
proposed  design  can  meet  those  requirements  within  cost,  schedule, 
and  other  system  constraints.  We  have  also  found  that  programs 
conducting  these  events  prior  to  development  start  experienced  less 
research  and  development  cost  growth  and  shorter  delays  in  the 
delivery  of  initial  operational  capabilities  than  programs  that 
conducted  these  reviews  after  development  start.11  Almost  all  nonship 
programs  (37  of  40  that  provided  data)  in  our  latest  assessment  have 
held  at  least  one  of  three  key  systems  engineering  reviews  (system 
requirements  review,  system  functional  review,  and  preliminary  design 
review).  However,  only  1  of  37  programs  that  held  a  preliminary  design 
review  did  so  before  the  start  of  system  development.  The  remaining 
programs  held  the  review,  on  average,  30  months  after  development 
start.  The  Weapon  Systems  Acquisition  Reform  Act  of  2009  established 


10A  major  defense  acquisition  program  may  not  receive  milestone  B  approval  until  the 
milestone  decision  authority  certifies  that  the  technology  in  the  program  has  been 
demonstrated  in  a  relevant  environment.  National  Defense  Authorization  Act  for  Fiscal 
Year  2006,  Pub.  L.  No.  109-163,  §  801  (codified  at  10  U.S.C.  §  2366b  (a)(3)(D)). 

nGAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  GAO-09-326SP, 
Washington,  D.C.:  March  30,  2009. 
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a  statutory  requirement  for  programs  to  conduct  a  preliminary  design 
review  before  milestone  B,  so  we  expect  improvements  in  this  area.12 

•  Programs  that  have  held  critical  design  reviews  in  recent  years 
reported  higher  levels  of  design  knowledge;  however,  few 
programs  are  demonstrating  that  the  design  is  capable  of 
meeting  performance  requirements  by  testing  an  integrated 
prototype.  Knowing  a  product’s  design  is  stable  before  system 
demonstration  reduces  the  risk  of  costly  design  changes  occurring 
during  the  manufacturing  of  production-representative  prototypes — 
when  investments  in  acquisitions  become  more  significant.  The  overall 
design  knowledge  that  programs  have  demonstrated  at  their  critical 
design  reviews  has  increased  since  2003.  Programs  in  our  assessment 
that  held  a  critical  design  review  between  2006  and  2009  had,  on 
average,  almost  70  percent  of  their  design  drawings  releasable  at  the 
time  of  the  review,  which  is  a  consistent  upward  trend  since  2003. 
However,  most  designs  are  still  not  stable  at  this  point.  Of  the  28 
programs  in  our  latest  assessment  that  have  held  a  system-level  critical 
design  review,  only  8  reported  having  a  stable  design.  Only  2  of  the  5 
programs  that  held  a  critical  design  review  in  2009  had  a  stable  design 
at  that  point.  The  5  programs  reported  that,  on  average,  83  percent  of 
the  total  expected  drawings  were  releasable. 

While  the  design  knowledge  of  DOD  programs  at  the  system-level 
critical  design  review  has  increased  since  2003,  these  programs  are  still 
not  regularly  demonstrating  that  these  designs  can  meet  performance 
requirements  by  testing  integrated  prototypes  before  the  critical  design 
review — a  best  practice.  None  of  the  5  programs  in  our  latest 
assessment  that  held  their  critical  design  review  in  2009  and  planned  to 
test  a  prototype  did  so  before  the  review.  Of  the  33  programs  that 
reported  that  they  either  had  tested  or  were  going  to  test  an  early 
system  prototype  and  provided  a  critical  design  review  date,  only  4  did 
so  before  their  critical  design  review.13  The  Weapon  Systems 


12Under  the  Weapon  Systems  Acquisition  Reform  Act  of  2009,  a  major  defense  acquisition 
program  may  not  receive  milestone  B  approval  until  the  program  has  held  a  preliminary 
design  review  and  the  milestone  decision  authority  has  conducted  a  formal  post¬ 
preliminary  design  review  assessment  and  certified  on  the  basis  of  such  assessment  that 
the  program  demonstrates  a  high  likelihood  of  accomplishing  its  intended  mission.  Pub.  L. 
No.  111-23,  §  205(a)(3)  (codified  as  amended  at  10  U.S.C.  §  2366b(a)(2)). 

13One  program  that  held  a  critical  design  review  in  2009  did  not  plan  to  test  an  early 
systems  prototype. 
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Acquisition  Reform  Act  of  2009  requires  that  DOD  policy  ensure  that 
the  acquisition  strategy  for  each  major  defense  acquisition  program 
provides  for  competitive  prototypes  before  milestone  B  approval, 
unless  a  waiver  is  properly  granted.14  This  requirement  should  increase 
the  percentage  of  programs  demonstrating  that  the  system’s  design 
works  as  intended  before  the  critical  design  review. 

•  Some  programs  are  taking  steps  to  bring  critical  manufacturing 
processes  into  control,  however  many  programs  still  rely  on  “after 
the  fact”  metrics.  Capturing  critical  manufacturing  knowledge  before 
entering  production  helps  ensure  that  a  weapon  system  will  work  as 
intended  and  can  be  manufactured  efficiently  to  meet  cost,  schedule, 
and  quality  targets.  Identifying  key  product  characteristics  and  the 
associated  critical  manufacturing  processes  is  a  key  initial  step  to 
ensuring  production  elements  are  stable  and  in  control.  Seven 
programs  in  our  latest  assessment  have  identified  their  critical 
manufacturing  processes,  including  four  of  the  programs  that  entered 
production  in  2009.  Three  of  those  seven  programs  reported  that  their 
critical  manufacturing  processes  were  in  control.15  It  is  generally  less 
costly — in  terms  of  time  and  money — to  eliminate  product  variation  by 
controlling  manufacturing  processes  than  to  perform  extensive 
inspection  after  a  product  is  built.  However,  many  DOD  programs  rely 
on  inspecting  produced  components  instead  of  using  statistical  process 
control  data  in  order  to  assess  the  maturity  of  their  production 
processes.  For  example,  12  programs  in  our  assessment  reported 
tracking  defects  in  delivered  units,  nonconformances,  or  scrap/rework 
as  a  way  to  measure  production  process  maturity.  The  use  of  “after  the 
fact”  metrics  is  a  reactive  approach  towards  managing  manufacturing 
quality  as  opposed  to  a  prevention-based  approach. 

•  Programs  are  still  not  regularly  testing  production 
representative  prototypes  before  committing  to  production.  We 

have  previously  reported  that  in  addition  to  demonstrating  that  the 


14Pub.  L.  No.  111-23,  §  203. 

15DOD  policy  states  that  the  knowledge  required  for  a  major  defense  acquisition  program  to 
proceed  beyond  low-  rate  initial  production  shall  include  demonstrated  control  of  the 
manufacturing  process  and  acceptable  reliability,  the  collection  of  statistical  process 
control  data,  and  demonstrated  control  and  capability  of  critical  processes.  Department  of 
Defense  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System,  enclosure  2, 
paragraph  7.c.(2)  (Dec.  8,  2008).  We  did  not  specifically  assess  compliance  with  this 
requirement. 
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system  can  be  built  efficiently,  production  and  postproduction  costs 
are  minimized  when  a  fully  integrated,  capable  prototype  is 
demonstrated  to  show  that  the  system  will  work  as  intended  and  in  a 
reliable  manner.  The  benefits  of  testing  are  maximized  when  the  tests 
are  completed  prior  to  a  production  decision  because  making  design 
changes  after  production  begins  can  be  both  costly  and  inefficient. 
However,  of  the  32  programs  in  our  assessment  that  could  have  tested 
a  prototype  before  production,  only  17  either  tested  or  expect  to  test  a 
fully  configured,  integrated,  production-representative  prototype 
before  holding  their  production  decision.  In  December  2008,  DOD 
changed  its  policy  to  require  programs  to  test  production- 
representative  articles  before  entering  production. 

•  More  programs  are  using  reliability  growth  curves  before 
beginning  production.  Reliability  growth  testing  provides  visibility 
over  how  reliability  is  improving  and  uncovers  design  problems  so 
fixes  can  be  incorporated  before  production  begins.  According  to 
DOD’s  acquisition  policy,  a  major  defense  acquisition  program  may  not 
proceed  beyond  low-rate  initial  production  until  it  has  demonstrated 
acceptable  reliability.  Over  half — 22  of  40  programs  that  responded  to 
our  questionnaire — reported  that  they  use  a  reliability  growth  curve, 
with  18  of  these  programs  reporting  they  are  currently  meeting  their 
established  goals.  In  addition,  12  of  19  programs  that  expect  to  hold 
their  production  decision  in  2010  and  beyond  reported  using  reliability 
growth  curves  and  most  stated  they  are  currently  meeting  their  goals. 
This  practice  should  help  these  programs  begin  production  with  a 
reliable  product  design. 


Observations  on 
Other  Factors  That 
Can  Affect  Program 
Execution 


Our  2010  assessment  of  weapon  programs  also  included  three 
observations  on  other  areas  related  to  DOD’s  management  of  its  weapons 
programs,  including  requirements,  software  management,  and  program 
office  staffing.  We  have  previously  identified  requirements  changes  and 
increases  in  software  lines  of  code  as  sources  of  program  instability  that 
can  contribute  to  cost  growth  and  schedule  delays.  We  have  also  reported 
that  workforce  challenges  can  hinder  program  execution  and  negatively 
affect  program  management  and  oversight. 

•  A  majority  of  programs  changed  key  systems  requirements  after 
development  start.  Of  the  42  programs  in  our  2010  assessment  that 
reported  tracking  requirements  changes,  23  programs  reported  having 
had  at  least  one  change  (addition,  reduction,  enhancement,  or 
deferment)  to  a  key  performance  parameter — a  top-level 
requirement — since  development  start.  Further,  nine  programs 
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experienced  at  least  one  change  to  a  key  system  attribute — a  lower 
level,  but  still  a  crucial  requirement  of  the  system.  Eight  programs 
reported  major  effects  on  the  program  as  a  result  of  these  requirements 
changes,  such  as  not  meeting  acquisition  program  baseline  cost, 
schedule,  and  performance  thresholds.  DOD’s  revised  December  2008 
acquisition  policy  attempts  to  reduce  potentially  disruptive 
requirements  changes  by  requiring  programs  to  hold  annual 
configuration  steering  board  meetings  to  ensure  that  significant 
technical  changes  are  not  approved  without  considering  their  effect  on 
cost  and  schedule. 

•  Many  programs  are  at  risk  for  cost  growth  and  schedule  delays 
because  of  software  development  issues.  Seventeen  of  the  28 
programs  in  our  2010  assessment  that  reported  data  on  software  lines 
of  code  estimated  that  the  number  of  lines  of  code  required  for  the 
system  to  function  has  grown  or  will  grow  by  25  percent  or  more — a 
predictor  of  future  cost  and  schedule  growth.  Overall,  the  average 
growth  or  expected  growth  in  lines  of  code  for  the  28  programs  was 
about  92  percent.  In  addition  to  measuring  growth  in  software  lines  of 
code,  we  have  previously  reported  that  collecting  earned  value 
management  data  for  software  development  and  tracking  and 
containing  software  defects  in  phase  are  good  management  practices. 
Overall,  30  programs  in  our  assessment  reported  collecting  earned 
value  management  data  to  help  manage  software  development.  Thirty- 
two  programs  in  our  latest  assessment  also  reported  collecting  some 
type  of  software  defect  data.  For  the  22  programs  that  responded  a 
more  specific  question  about  defect  correction,  on  average,  only  69 
percent  of  the  defects  were  corrected  in  the  phase  of  software 
development  in  which  they  occurred.  Capturing  software  defects  in 
phase  is  important  because  discovering  defects  out  of  phase  can  cause 
expensive  rework  later  in  programs. 

•  Programs’  reliance  on  nongovernment  personnel  continues  to 
increase  in  order  to  make  up  for  shortfalls  in  government 
personnel  and  capabilities.  In  recent  years,  Congress  and  DOD  have 
taken  steps  to  ensure  the  acquisition  workforce  has  the  capacity, 
personnel,  and  skills  needed  to  properly  perform  its  mission;  however, 
programs  continue  to  struggle  to  fill  all  staff  positions  authorized.  Only 
19  of  the  50  programs  in  our  2010  assessment  that  responded  to  our 
questions  on  staffing  were  able  to  fill  all  the  positions  they  had  been 
authorized.  A  commonly  cited  reason  for  not  being  able  to  fill  positions 
was  difficulty  finding  qualified  candidates.  As  a  result  of  staff  shortfalls, 
program  offices  reported  that  program  management  and  oversight  has 
been  degraded,  contracting  activities  have  been  delayed,  and  program 
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management  costs  have  increased  as  contractors  are  used  to  fill  the 
gap.  Overall,  43  programs  or  86  percent  of  those  providing  data 
reported  utilizing  support  contractors  to  make  up  for  shortfalls  in 
government  personnel  and  capabilities. 

In  addition,  for  the  first  time  since  we  began  reporting  on  program 
office  staffing  in  2008,  programs  reported  having  more  nongovernment 
than  government  staff  working  in  program  offices  (see  table  2).  The 
greatest  numbers  of  support  contractors  are  in  engineering  and 
technical  positions,  but  their  participation  has  increased  in  all  areas, 
from  program  management  and  contracting  to  administrative  support 
and  other  business  functions. 


Table  2:  Program  Office  Composition  for  50  DOD  Programs 


Percentage  of  staff 

Other 


Program 

management 

Engineering  and 
technical 

Contracting 

business 

functions 

Administrative 

support 

Other 

Total 

Military 

28 

7 

6 

3 

2 

5 

8 

Civilian  government 

40 

41 

74 

45 

18 

24 

40 

Total  government 

67 

47 

80 

48 

20 

29 

49 

Support  contractors 

32 

43 

20 

50 

78 

70 

45 

Other  nongovernment1 

0 

9 

0 

3 

2 

1 

6 

Total  nongovernment 

33 

53 

20 

52 

80 

71 

51 

Source:  GAO  analysis  of  DOD  data. 

Notes:  Totals  may  not  add  due  to  rounding. 

“Other  nongovernment  includes  federally  funded  research  and  development  centers,  universities,  and 
affiliates. 


Observations  about 
DOD’s 

Implementation  of 
Acquisition  Reforms 


DOD  has  begun  to  incorporate  acquisition  reforms  into  the  acquisition 
strategies  for  new  programs.  Both  DOD’s  December  2008  acquisition 
policy  revisions  and  the  Weapon  Systems  Acquisition  Reform  Act  of  2009 
require  programs  to  invest  more  time  and  resources  in  the  front  end  of  the 
acquisition  process — refining  concepts  through  early  systems  engineering, 
developing  technologies,  and  building  prototypes  before  starting  system 
development.  In  addition,  DOD  policy  requires  establishment  of 
configuration  steering  boards  that  meet  annually  to  review  all  program 
requirements  changes  as  well  as  to  make  recommendations  on  proposed 
descoping  options  that  could  help  keep  a  program  within  its  established 
cost  and  schedule  targets.  These  steps  could  provide  a  foundation  for 
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establishing  sound,  knowledge-based  business  cases  for  individual 
weapon  programs  and  are  consistent  with  many  of  our  past 
recommendations;  however,  if  reform  is  to  succeed  and  weapon  program 
outcomes  are  to  improve,  they  must  continue  to  be  reinforced  in  practice 
through  decisions  on  individual  programs. 

Our  analysis  of  the  programs  in  our  2010  assessment  allowed  us  to  make 
two  observations  about  the  extent  to  which  DOD  is  implementing  recent 
acquisition  reforms: 

•  Most  of  the  ten  programs  in  our  2010  assessment  that  had  not 
yet  entered  system  development  reported  having  acquisitions 
strategies  consistent  with  both  DOD’s  revised  acquisition  policy 
and  the  provisions  of  the  Weapon  Systems  Acquisition  Reform 
Act  of  2009.  Specifically,  8  programs  in  our  assessment  planned  to 
develop  competitive  prototypes  before  milestone  B.16  In  addition,  7 
programs  have  already  scheduled  a  preliminary  design  review  before 
milestone  B.17 

•  Only  a  few  programs  reported  holding  configuration  steering 
boards  to  review  requirements  changes,  significant  technical 
changes,  or  de-scoping  options  in  2009.  Seven  programs  in  our 
assessment  reported  holding  configuration  steering  boards  in  2009. 
Under  DOD’s  revised  acquisition  policy,  ongoing  acquisition  category  I 
and  LA.  programs  in  development  are  required  to  conduct  annual 
configuration  steering  boards  to  review  requirements  changes  and 
significant  technical  configuration  changes  that  have  the  potential  to 
result  in  cost  and  schedule  effects  on  the  program.  In  addition,  the 
program  manager  is  expected  to  present  de-scoping  options  to  the 
board  that  could  reduce  program  costs  or  moderate  requirements. 
None  of  the  programs  reported  that  the  boards  that  were  held 
approved  requirements  changes  or  significant  technical  changes.  One 


16The  Weapon  Systems  Acquisition  Reform  Act  of  2009  requires  that  DOD  policy  ensure 
that  the  acquisition  strategy  for  each  major  defense  acquisition  program  provides  for 
competitive  prototypes  before  milestone  B  approval,  unless  a  waiver  is  properly  granted. 
Pub.  L.  No.  111-23,  §  203(a). 

1  The  Weapon  Systems  Acquisition  Reform  Act  of  2009  establishes  a  statutoiy  requirement 
that  a  major  defense  acquisition  program  may  not  receive  milestone  B  approval  until  the 
milestone  decision  authority  has  received  a  preliminary  design  review,  conducted  a  formal 
post-preliminary  design  review  assessment,  and  certified  on  the  basis  of  such  assessment 
that  the  program  demonstrates  a  high  likelihood  of  accomplishing  its  intended  mission. 
Pub.  L.  No.  111-23,  §  205(a)(3)  (codified  as  amended  at  10  U.S.C.  §  2366b(a)(2)). 
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program — the  P-8A  Poseidon — reported  that  it  presented  de-scoping 
options  to  decrease  cost  and  schedule  risk  on  the  program  and  had 
those  options  approved. 


Concluding 
Observations  on  the 
Challenges  to 
Achieving  Lasting 
Reform 


I  would  like  to  offer  a  few  thoughts  about  other  factors  that  should  be 
considered  so  that  we  make  the  most  out  of  today’s  opportunity  for 
meaningful  change.  First,  I  think  it  is  useful  to  think  of  the  processes  that 
affect  weapon  system  outcomes  (requirements,  funding,  and  acquisition) 
as  being  in  a  state  of  equilibrium.  Poor  outcomes — delays,  cost  growth, 
and  reduced  quantities — have  been  persistent  for  decades.  If  we  think  of 
these  processes  as  merely  “broken”,  then  some  targeted  repairs  should  fix 
them.  I  think  the  challenge  is  greater  than  that.  If  we  think  of  these 
processes  as  being  in  equilibrium,  where  their  inefficiencies  are  implicitly 
accepted  as  the  cost  of  doing  business,  then  the  challenge  for  getting 
better  outcomes  is  greater.  Seen  in  this  light,  it  will  take  considerable  and 
sustained  effort  to  change  the  incentives  and  inertia  that  reinforce  the 
status  quo. 


Second,  while  actions  taken  and  proposed  by  DOD  and  Congress  are 
constructive  and  will  serve  to  improve  acquisition  outcomes,  one  has  to 
ask  the  question  why  extraordinary  actions  are  needed  to  force  practices 
that  should  occur  normally.  The  answer  to  this  question  will  shed  light  on 
the  cultural  or  environmental  forces  that  operate  against  sound 
management  practices.  For  reforms  to  work,  they  will  have  to  address 
these  forces  as  well.  For  example,  there  have  been  a  number  of  changes  to 
make  cost  estimates  more  rigorous  and  realistic,  but  do  these  address  all 
of  the  reasons  why  estimates  are  not  already  realistic?  Clearly,  more 
independence,  methodological  rigor,  and  better  information  about  risk 
areas  like  technology  will  make  estimates  more  realistic.  On  the  other 
hand,  realism  is  compromised  as  the  competition  for  funding  encourages 
programs  to  appear  affordable.  Also,  when  program  sponsors  present  a 
program  as  more  than  a  weapon  system,  but  rather  as  essential  to  new 
fighting  concepts,  pressures  exist  to  accept  less  than  rigorous  cost 
estimates.  Reform  must  recognize  and  counteract  these  pressures  as  well. 


Third,  decisions  on  individual  systems  must  reinforce  good  practices. 
Programs  that  have  pursued  risky  and  unexecutable  acquisition  strategies 
have  succeeded  in  winning  approval  and  funding.  If  reform  is  to  succeed, 
then  programs  that  present  realistic  strategies  and  resource  estimates 
must  succeed  in  winning  approval  and  funding.  Those  programs  that 
continue  past  practices  of  pushing  unexecutable  strategies  must  be  denied 
funding  before  they  begin.  This  will  require  sustained  leadership  from  the 
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Secretary  of  Defense,  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics,  and  the  military  services,  and  the  cooperation 
and  support  of  Congress. 

Fourth,  consideration  should  be  given  to  setting  some  limits  on  what  is  a 
reasonable  length  of  time  for  developing  a  system.  For  example,  if  a 
program  has  to  complete  development  within  5  or  6  years,  this  could  serve 
as  a  basis  to  constrain  requirements  and  exotic  programs.  It  would  also 
serve  to  get  capability  in  the  hands  of  the  warfighter  sooner. 

Fifth,  the  institutional  resources  we  have  must  match  the  outcomes  we 
desire.  For  example,  if  more  work  must  be  done  to  reduce  technical  risk 
before  development  start — milestone  B — DOD  needs  to  have  the 
organizational,  people,  and  financial  resources  to  do  so.  Once  a  program  is 
approved  for  development,  program  offices  and  testing  organizations  must 
have  the  workforce  with  the  requisite  skills  to  manage  and  oversee  the 
effort.  Contracting  instruments  must  be  used  that  match  the  needs  of  the 
acquisition  and  protect  the  government’s  interests.  Finally,  DOD  must  be 
judicious  and  consistent  in  how  it  relies  on  contractors. 


Mr.  Chairman,  this  completes  my  prepared  statement.  I  would  be  happy  to 
respond  to  any  questions  you  or  other  Members  of  the  Subcommittee  may 
have  at  this  time. 
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